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PROFILING2  THE  CHARACTERISTICS  OF  THE  DEVELOPMENTAL 

TEST  PARTICIPANT 


INTRODUCTION 

The  data  from  developmental  tests  (DT)  describing  system*5  performance  and  system 
reliability  often  do  not  agree  with  data  from  operational  tests  (OT)  conducted  on  the  same 
system.  The  difference  can  originate  from  any  combination  of  a number  of  sources  which  can  be 
broadly  categorized  as  (a)  changes  in  the  hardware  being  tested,  (b)  differences  in  the  test 
methodology  and  conditions,  and  (c)  the  personnel  who  participated  as  users  or  operators  of  the 
hardware. 

Changes  in  the  hardware  involved  in  the  system  might  include  wear,  different  combinations 
or  configurations  of  components,  or  perhaps  "fixes”  which  may  have  been  implemented  between 
DT  and  OT.  Differences  due  to  wear  can  be  addressed  by  physical  measurement  of  tolerances  and 
documented  by  the  amount  of  use  the  hardware  has  received.  Different  combinations  or 
configurations  of  components  should  be  a routine  part  of  the  description  of  the  equipment  in  a 
system  test.  Any  repairs  or  modifications  of  the  equipment  should  also  routinely  receive 
thorough  documentation.  In  short,  although  changes  in  the  hardware  would  certainly  be 
expected  to  impact  on  performance  and  reliability  data,  the  changes  themselves  should  not  be 
difficult  to  identify. 

The  test  conditions  and  methodology  will  almost  always  be  different  in  DT  and  in  OT,  but 
are  the  subject  of  detailed  description  in  the  test  plan.  Procedures  and  guidelines  for  their 
implementation  already  exist  for  describing  differences  in  test  conditions  and  methods  which 
might  affect  test  results. 


THE  PROBLEM 

The  third  category  of  differences-the  influence  of  the  human  operator-has  not  been 
adequately  addressed.  At  the  present  time  there  is  no  procedure  built  into  the  testing 
methodology  for  examining  this  source  of  variation.  It  is  this  gap  in  the  methodology  which  is 
addressed  in  this  proposal. 


aThe  term  “profile”  is  used  by  the  military  in  several  different  ways.  The  soldier's  personnel 
records  use  it  to  describe  a set  of  ratings  of  physical  attributes.  This  usage  refers  to  a physical 
profile  (4).  If  the  soldier  has  an  injury  or  illness,  the  term  special-duty  profile  specifies  activities 
in  which  the  soldier  should  not  engage.  Unless  otherwise  stated,  the  term  "profile”  as  it  appears 
in  this  report  has  a broader  meaning  than  either  of  these  usages.  It  refers  to  a set  of  physical  and 
intellectual  characteristics  which  may  affect  performance  in  system  tests. 


bThe  term  “system”  as  used  throughout  this  report  should  be  understood  to  include  the  human 
operator. 
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PROPOSED  SOLUTION 

Because  human  performance  influences  the  total  system’s  performance,  we  need  methods  to 
control  the  variance  contributed  by  the  human  operator.  The  most  direct  method  of  control  is 
personnel  selection.  If  participants  in  tests  are  selected  according  to  criteria  which  eliminate 
personnel  who  fall  at  extreme  ends  of  the  distributions  of  values  of  relevant  characteristics, 
variation  in  performance  should  be  reduced  accordingly.  If  this  variation  in  performance  has  the 
effect  of  noise  (in  an  information-theory  sense)  on  measurements  of  system  performance  or 
reliability,  reducing  this  “noise”  should  enhance  our  measurement  procedures’  sensitivity  to  the 
sources  of  variation  which  are  the  real  subject  of  the  test. 

Once  the  sample  of  participants  has  been  screened  according  to  the  selection  criteria,  the 
selected  participants  should  be  described  in  greater  detail  by  means  of  biographical  data.  The 
collection  of  biographical  data  is  not,  in  itself,  a control  on  human  variation.  The  benefit  which 
accrues  from  this  approach  is  that  the  evaluator  has  an  a posteriori  method  of  accounting  for 
differences  which  arc  not  attributable  to  other  sources  of  variation. 

A longer-run  benefit  from  the  proposed  approach  would  be  development  of  performance- 
based  selection  criteria.  If  a data  base  is  built  up  consisting  of  characteristic  descriptive  data  on 
test  participants  and  the  level  of  performance  they  attain  on  specific  tasks  in  testing,  by  reviewing 
the  correlation  between  task  performance  and  descriptive  data  across  participants  we  should  be 
able  to  develop  a profile  of  levels  of  these  characteristics  which  are  required  by  similar  tasks  in 
other  systems.  In  addition  to  selecting  test  participants,  the  profile  might  also  alert  the  evaluator 
when  the  system  hardware  requires  a profile  of  operator  characteristics  which  is  unusual.  In  the 
early  stages  of  development  this  information  could  serve  as  a check  on  system  engineering  and 
could  identify  problems  at  a stage  of  development  when  it  is  relatively  inexpensive  to  make 
remedial  changes  in  hardware  design. 


Selection  Criteria 

The  basic  purpose  of  the  selection  criteria  is  to  eliminate  extreme  values  of  the  selection 
characteristics.  Although  it  would  be  desirable  to  have  the  characteristics  of  the  sample  of  test 
participants  distributed  in  the  same  proportions  as  in  the  population  of  interest  on  each  variable, 
the  criteria  proposed  here  would  serve  only  to  reduce  the  sample  variance.  The  requirements  for 
the  test  participant  should  include  specification  of  his  required  military  training,  his  basic 
physical  dimensions,  and  his  perceptual  abilities. 


Training  Requirements 

The  required  MOS,  complete  with  skill-level  suffix,  should  be  specified.  This 
specification  should  include  a determination  of  whether  the  specified  MOS  must  be  the 
participant’s  primary  MOS  or  whether  a soldier  with  this  specialty  in  a secondary  MOS  is 
acceptable.  If  alternate  specialties  include  the  required  training  and  are  acceptable  substitutes, 
these  should  also  be  listed.  In  addition,  the  training-requirements  specifications  should  state 
whether  the  MOS  must  be  a school-trained  qualification  or  whether  an  OJT-qualified  soldier 
meets  the  requirements. 
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Physical  Dimensions 

Minima  and  maxima  for  height  and  weignt  should  be  specified,  giving  due 
consideration  to  the  range  of  these  dimensions  expected  in  typical  users  when  the  system  is 
fielded.  One  approach  might  be  to  establish  centile  cutoff  scores  first  for  height,  then  determine 
upper  and  lower  centile  cutting  scores  for  weight  at  each  height.  A graphic  example  of  how  this 
might  be  done  is  presented  in  the  Appendix. 

No  personnel  with  a special-duty  profile  should  be  permitted  to  participate  unless  a 
task  analysis  reveals  that  the  restriction  on  his  activities  has  no  impact  on  the  tasks  required  in 
the  test. 


Sensory  Acuity 

All  participants  should  have  had  a recent  (within  the  last  12  months)  test  of  vision  and 
audition.  Minimum  standards  should  be  stated  for  each  of  these  sensory  modalities  depending 
upon  an  analysis  of  the  requirements  of  the  tasks  to  be  performed.  A good  starting  point  for 
standards  might  be  those  required  of  new  inductee-  into  the  Army.  Although  there  is  room  here 
for  the  application  of  judgment  in  individual  tests,  standards  should  nonetheless  be  established 
and  stated, together  with  their  rationale. 


Descriptive  Data 

After  participants  have  been  selected  according  to  criteiL,  a more  detailed  description  of 
their  characteristics  should  be  obtained  by  collecting  the  following  information  on  each  test 
participant: 

1.  Actual  value  for  the  participant  on  each  of  the  selection  criterion  measurements 

2.  Scores  from  aptitude  tests  administered  when  the  participant  first  entered  the  Army. 

3.  Scores  from  the  participants’  most  recent  MOS  tests. 

4.  Experience  (in  months)  in  the  relevant  MOS. 

5.  Total  service  time. 

6.  A list  (by  descriptive  title)  of  additional  system-related  training  or  courses 
completed. 

7.  Results  of  tests  administered  during  system-specific  training. 

8.  Performance  criteria  applied  during  system-specific  training,  including: 

a.  Minimum  performance  required,  and 

b.  Level  of  performance  attained  by  the  participant. 

9.  Civilian  Education.  Both  the  level  attained  and  the  specialty,  if  any,  should  be 

recorded. 
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The  above  items  of  information  can  be  collected  easily.  Most  of  them  are  available  from  the 
participants’  personnel  files.  Item  one  can  be  recorded  for  each  participant  as  a part  of  the 
selection  process.  Items  two  through  six  should  be  available  from  personnel  files.  Items  seven  and 
eight  should  be  a part  of  the  system-specific  training  and  can  be  collected  at  that  time.  Item  nine 
may  be  up-to-date  in  personnel  files,  but  this  and  other  pertinent  administrative  data  can  be 
collected  or  confirmed  via  a simple  data-collection  form  administered  on  the  first  day  of  the  test. 

While  the  relationship  between  each  of  the  first  eight  items  and  soldier  performance  should 
be  apparent,  item  nine  is  included  because  the  education  the  participant  brought  with  him  into 
the  Army  may  influence  his  performance.  It  would  be  useful  to  know,  for  example,  whether  the 
operator  of  a fire-direction  center  had  a strong  background  in  mathematics,  drafting,  computer 
science,  or  other  related  fields. 


It 


DISCUSSION 

For  the  developmental  tester  and  evaluator,  one  important  benefit  of  this  approach  would 
be  greater  face  validity  of  the  test  data.  Because  the  selection  of  participants  for  operational  tests 
often  involves  simply  specifying  a unit  type,  then  allowing  a U.S.  Army  Forces  Command 
commander  to  select  such  a unit  from  the  target  population,  the  troops  who  participate  in 
operational  tests  have  some  built-in  believability:  they  are  sampled  (not  randomly,  and  probably 
not  representatively,  however)  from  the  user  population.  Participants  in  developmental  tests 
often  include  civilian  technicians  in  early  stages  of  development.  Soldiers  who  participate  are 
selected  on  an  individual  basis  and  may  be  more  experienced  in  DT  participation  than  in  the 
relevant  MOS.  Since  these  individually  designated  participants  are  not  selected  because  they  are 
members  of  a user-type  unit  (by  definition,  part  of  the  user  population)  but  are  selected 
individually,  they  do  not  have  the  built-in  believability  enjoyed  by  their  counterparts  in 
operational  tests.  Thus,  when  DT  and  OT  project  different  performance  and  reliability  values,  the 
operational  tester  appears  to  argue  from  a position  of  strength.  If  DT  participants  can  be  selected 
in  a way  that  eliminates  the  extremes  of  the  distribution,  and  if  biographical  data  are  available  to 
describe  the  population  from  which  they  were  sampled,  suggestions  that  the  differences  between 
DT  and  OT  are  due  to  "test-wise”  DT  participants  loses  some  of  its  credibility. 

It  would  not  be  practical  and  might  not  even  be  desirable  to  try  to  select  OT  participants 
according  to  the  same  scheme.  However,  the  comparability  of  test  data  between  OT  and  DT 
could  be  enhanced  by  collecting  biographical  data  on  OT  participants  similar  to  that  proposed 
above  for  DT  participants. 

The  purposes  of  DT  and  OT  are  understandably  different.  As  mentioned  earlier,  the  test 
conditions  and  methodology  will  almost  always  be  different.  But  if  we  include  the  human 
operator  in  the  system  concept,  to  test  one  time  using  civilian  technicians  and  test-wise  soldiers, 
then  again  using  participants  from  the  intended  population,  is  to  test  two  different  systems.  The 
utility  of  demonstrating  that  these  two  systems  behave  and  perform  differently  is  not 
compellingly  apparent. 

The  concept  of  selection  and  description  of  test  participants  is  not  new.  It  has  been  applied 
in  testing  on  a somewhat  unsystematic  basis  for  several  years.  The  U.S.  Army  Test  and  Evaluation 
Command  (TECOM)  includes  in  many  of  its  Materiel  Test  Procedures  (MTPs)  and  Test 
Operations  Procedures  (TOPs)  provisioy  for  recording  descriptive  data  on  test  participants. 
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In  MTPs  on  human  factors  engineering  (8)  there  are  instructions  concerning  selection  of  test 
participants:  “Make  certain  that  the  operators  used  in  the  evaluation  represent  those  who  will  be 
operating  the  equipment  in  actual  use,  particularly  witl.  respect  to  such  characteristics  as  age, 
general  ability,  experience  and  training.”  This  is  followed  by  a series  of  guidelines  which  include 
avoiding  biased  subjects  and  “expert  operators",  and  recommendations  that  the  tester  “motivate 
the  operators  to  the  same  extent  they  are  likely  to  be  motivated  in  future  use  of  the  system. ”c 

Contractor  Demonstration  (CD)  testing  (5)  often  includes  attempts  to  select  and  to  describe 
test  participants.  Since  January  of  1972,  contractors  have  been  required  (2)  to  include  in  their 
reports  of  human  factors  engineering  tests  a description  of  test  participants.  The  most  recent 
description  requirement  (3)  includes  recording  for  each  participant  age,  weight,  body  dimensions 
relevant  to  performance  tasks,  visual  acuity  and  hearing  levels,  any  known  physical  disabilities, 
and  score  from  a standardized  measure  of  general  intelligence.  Tests  conducted  jointly  with 
foreign  governments  have  included  similar  requirements  (6).  Earlier  applications  of  the  concept 
include  selection  of  personnel  for  training  as  rangefinder  operators  back  in  1942  (7). 

The  point  to  be  emphasized  is  that  the  present  proposal  does  not  represent  a radical  change 
in  current  materiel-testing  procedures.  Rather,  it  suggests  that  we  impose  some  consistency  upon 
procedures  which  previously  have  been  followed  with  varying  degrees  of  thoroughness.  The 
Appendix  gives  an  example  of  how  the  profile  criteria  might  be  applied  and  the  descriptive  data 
collected. 

In  conclusion,  it  is  proposed  that  selecting  developmental-test  participants  according  to 
uniformly  applied  criteria,  then  describing  in  greater  detail  the  population  from  which  they  were 
selected,  offers  a means  of  controlling  the  system  performance  variation  contributed  by  the 
human  operator.  The  effort  required  to  implement  this  proposal  appears  small  when  compared  to 
the  benefits  to  be  gained  in  the  form  of  reduced  variation  between  samples  of  operators  and  of 
enhanced  accuracy  in  predicting  system  effectiveness  in  the  field. 


cThe  quoted  MTP  is  one  of  several  which  are  concerned  with  human  factors  engineering.  In 
addition  to  the  personnel  selection  and  description  guidance  in  the  human  factors  MTPs,  many 
other  MTPs  and  TOPs  addressing  specific  classes  of  systems  include  lists  of  descriptive  data  which 
should  be  collected  on  developmental  test  participants. 
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The  following  application  of  the  proposed  selection  criteria  and  biographical  data  collection 
is  one  example  of  how  the  criteria  might  be  stated  and  the  data  obtained.  This  example  is  drawn 
from  a proposal  by  the  HEL  Detechment  (AMSAA)  for  selection  of  soldiers  to  participate  in 
testing  the  Main  Battle  Tank  XM-1 . It  is  not  intended  as  a general  format  to  be  followed  in  other 
tests,  because  each  testing  situation  presents  its  own  sets  of  requirements  and  constraints. 
However,  there  are  certain  facets  of  the  task  of  specifying  selection  criteria  which  are  common  to 
most  selection  exercises,  and  the  example  may  prove  useful  by  analogy. 


A.  The  specific  selection  criteria  for  SOMTE  personnel  are: 

1.  Training  Requirements 

Participants  must  have  a primary,  school-trained  MOS  of  11E,  with  at  least  6 months 
of  functional  experience  in  the  crew  position  assigned  for  the  test  (e.g., driver,  gunner,  loader, 
commander).  Participants’  skill-level  suffix  should  be  no  higher  than  the  minimum  required  by 
TOE  for  the  assigned  position. 

2.  Physical  Requirements 

Personnel  height  must  be  within  the  5th  to  the  95th  percentiles  for  height  of  armor 
crewmen.  Their  weight  must  be  within  the  5th  to  the  95th  percentiles  for  their  height.  Figure  1A 
illustrates  graphically  the  acceptable  height  and  weight  values.  In  addition,  the  participant  should 
have  a rating  of  “one”  in  each  of  the  six  factors  (PULHES)  of  the  physical  profile  serial  system 
(4)  unless  it  can  be  determined  from  a task  analysis  that  any  limitations  on  one  or  more  of  the 
factors  will  not  affect  test  results.  As  an  example,  a numerical  rating  lower  than  “one”  for  vision 
(E)  would  not  be  acceptable  in  a gunner,  driver  or  commander,  but  might  be  acceptable  in  a 
loader. 

3.  Perceptual  Requirements 

All  participants  should  have  had  a physical  examination  within  the  last  12  months, 
including  tests  of  vision  and  audition.  If  the  examination  data  are  older  than  12  months,  tests  of 
vision  and  audition  must  be  conducted  to  validate  the  profile  rating  for  PULHES  factors  H and  E. 


The  training-requirements  statement  allowed  only  a primary,  school-trained  MOS  of  11E 
and  6 months  of  functional  experience.  Note  that  the  skill-level  suffix  is  limited  to  the  lowest 
level  specified  by  TOE.  For  example,  the  driver  and  loader  must  have  11E10.  The  gunner  can  be 
no  higher  than  1 1 E20,  nor  can  the  commander  be  any  higher  than  11E30.  Although  this  may 
seem  unnecessarily  restrictive,  it  seems  unlikely  that  the  XM-1  system  will  operate  tactically  with 
/ a crew  that  is  highly  overqualified.  To  test  it  using  an  overqualified  crew  would  therefore  be  to 

test  a different  system. 


The  cutting  values  for  height  and  weight  (Figure  IA)  were  drawn  from  MIL-HDBK  759  (1) 
I.,  and  are  based  on  a 1966  sample  of  Army  men.  The  height  limits  for  the  5th  and  95th  percentiles 

were  established  first.  Then  the  5th  and  95th  percentile  weights  were  calculated  for  each  of  the 
t , extremes  of  the  acceptable  height  range.  The  four  points  thus  obtained  are  the  corners  of  the 

acceptable  height  and  weight  figure.  The  figure  would  include  about  81  percent  of  the  1966 
sample.  To  the  degree  that  the  population  of  Army  men  has  changed  in  height  and  weight  over 
the  past  10  years,  the  cutting  scores  used  here  are  inappropriate.  One  would  therefore  be  able  to 
present  valid  arguments  for  other  sets  of  cutting  scores.  The  present  scores  were  based  on  the 
latest  generally  available  data. 

| * 


B.  Descriptive  data  on  each  participant  may  be  collected  on  Form  1. 

The  descriptive  data  required  on  the  Personnel  Data  Form  are  divided  into  two  sections. 
Part  A includes  items  which  the  test  participant  can  provide  from  memory.  The  items  in  Part  B 
can  be  obtained  by  test-control  personnel  from  the  soldier’s  Personnel  Qualification  Record,  Part 
I (DA  Form  2)  and  Part  II  (DA  Form  2-1). 

If  personnel  data  are  retrievable  by  name  or  by  social  security  account  number,  as  in  items  1 
and  4 of  Form  1 , an  appropriate  privacy-act  disclosure  statement  should  be  prepared  according  to 
guidelines  in  OMB  Circular  No.  A-108,  which  defines  responsibilities  for  implementing  the 
Privacy  Act  of  1974  (Public  Law  No.  93-579,  5 U.S.C.  552a). 


* 
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FORM  1 

PERSONNEL  DATA  FORM 


A.  To  be  completed  by  test  participant. 

1.  NAME  2.  DATE 

3.  MOS  4.  ID  No.  

5.  Crew  Position  (in  the  test) 

6.  Months  of  experience  (in  tested  crew  position) 

7.  Height  8.  Weight 9.  Date  c 

10.  Length  of  service:  years, months 

11.  Civilian  Education:  (a)  1 2 3 4 5 6 7 8! 

14  15  16  (circle  number  of  years) 

b.  Major  area  (if  applicable): 

B.  To  be  completed  for  each  participant  by  test  control  personnel. 


9.  Date  of  birth 


_ months 
5 6 7 


12.  Physical  Profile:  PUL 

1 3.  Aptitude  Scores:  (a)  CO 

H E S 
(b)  FA 

- (c)  EL  _ 

— (d)  OF  _ 

(e)  GM 

(f)  MM 

.(g)  CL 

(h)  ST 

(i)  GT 

(j)  SC  

(k)  El 

.(1)  Gl 

(m)  Cl 

(n)  AD 

(o)  SK 

(P)  Tl 

_(q)  Al 

(r)  AP 

(s)  PA 

(t)  MK 

(u)  WK 

_(v)  AR 

(w)  MC  _ 

14.  Latest  MOS  test  score:  

15.  End-of-training  test  score: . 

16.  Minimum  performance  (a)  required: 

17.  List  of  military  schools  and  courses  completed: 


.(b)  attained: 


